Extending Surf to the Color Domain

Automatic extraction of local features from images plays an important role in many computer vision tasks. During the last years, much focus has been put on making the features invariant to geometric transformations such as a rotation and scaling of the image. Recently, some work has been published concerning the integration of color information into the detection and description step of SIFT. In various evaluations, it has been shown that including color information can increase distinctiveness and invariance to photometric transformations caused by illumination changes. In this paper we build on the results from these approaches and apply them to the SURF descriptor, which is advantageous compared to SIFT in terms of speed, making it a perfect candidate for online applications, for example in the field of robotics. Our results show significant improvements concerning the repeatability and destinctiveness of SURF for 3d objects under varying illumination directions. In contrast to many other evaluations we also determine the accuracy of the orientation assignment and include this into our comparisons. INTRODUCTION Although color cameras are widely spread nowadays, most popular state of the art feature detector and descriptor algorithms like SIFT [1] and SURF [2] still operate on intensity images only. It is obvious that by disregarding color values one loses information. An edge between a green and a blue patch for example would only provide the same information as between two shades of gray, other color edges might not be recognizable in a gray scale image at all. Additional color information could be useful in the detection step to identify salient interest points defined by a change in color. A descriptor encoding color information is expected to provide a higher distinctiveness than those based on intensity only. From these assumptions the following questions arise: • How can color information be included in the detection to gain access to features whose localization is based on changes in color and not necessarily intensity? • How can color information be included in the descriptor to improve the distinctiveness and hence recognizability of feature points? In this paper we propose an extension to the SURF algorithm which works on color images and yields better results in terms of repeatability of the detector and distinctiveness of the descriptor. The rest of the paper is organized as follows: The next section will give an overview of lately proposed methods for enhancing existing feature descriptors with color information. Following that, we discuss basic choices of color space, the inclusion of color in scale space representations as well as color invariants and color boosting. The next two chapters deal with the integration of color information into the detection step and descriptor respectively. They also contain an evaluation of our algorithm. The final chapter summarizes the paper. Related Work During the recent years there were several suggestions how to include color information into state of the art local interest point detectors and descriptors. Goedemé et.al. [3] propose an additional step after matching SIFT descriptors in which a 3 dimensional color descriptor based on global color moments is used to sort out wrong matches, i.e. matches for which the distance of the color descriptors exceeds a fixed threshold. Bosch et.al.[4] introduce a color SIFT descriptor using HSV color space, but for a dense sampling approach in contrast to sparse interest point detection. Burghouts and Geusebroek [5] compare several SIFT descriptor variations utilizing color features invariant to different photometric transformations. Van de Weijer et.al. show how to improve the saliency of detected interest points by using color information [6] and propose a color histogram based method to improve the SIFT descriptor [7]. Abdel-Hakim and Farag [8] propose an extension to the detection as well as the descriptor step of SIFT based on color invariants. An overview and evaluation of possible color invariants for descriptors is given by van de Sande et.al. in [9]. Using color images In order to build a descriptor which is invariant to certain changes in illumination, we have to define the underlying reflectance model first. We assume Lambertian reflectance of surfaces and additional omnidirectional diffuse , which leads to an image creation process modeled as follows: Ik(x,y) = ∫ e(λ )s(x,y,λ )ρk(λ )dλ + ∫ a(λ )ρk(λ )dλ (1) e(λ ) are the spectral characteristics of the light source, s(x,y,λ ) is the surface reflectivity on the point measured by the sensor at (x,y) and ρk(λ ) is the camera sensitivity curve for channel k. a(λ ) is the ambient term. Possible color invariants Many photometric invariants have been proposed to gain robustness or invariance towards changes in illumination [10, 11, 12, 13, 6]. We take a closer look at two invariants proposed in [10] which delivered the best results in recent evaluations [9, 5]. Both are defined on the scale space representation Lk,k ∈ [1,2,3] of the image Ik: Lk(·,σ) = G(·,σ)∗ Ik , where G is a two-dimensional Gaussian kernel with variance σ , ∗ denotes the convolution and k the image channel. W invariant: The W invariant is defined as the derivative of the image signal, normalized by the intensity channel L1. It is invariant to local intensity changes, assuming planar surfaces which do not exhibit shading effects.